Move to ES bridge API #336

mashhurs · 2025-07-10T22:16:19Z

Description

This PR highlights the initial changes required to move to the Bridge Stable API. However, this doesn't mean these changes will be enough for the plugin to be a in normal working state. Rather, this changes highlight how does current bridge Spable API state align with plugin requirements. Based on the discussions on these changes, we may need to adjust the plan (sub-tasks).

Closes [Move to ES bridge -> ES scope] Evaluate current status of Stable API based on plugin usage #331

Next steps

Get review and land [Logstash] Move elastic_integration plugin usage to ES logstash-bridge. elasticsearch#131486
Adapt Java/Ruby unit test cases to move to bridge changes

PLEASE IGNORE:

gradle.build changes which are already counted in [Move to ES Bridge] Build logstash-bridge and include in the plugin #330, this is a requirement to call Bridge Stable API interfaces.

src/main/java/co/elastic/logstash/filters/elasticintegration/EventProcessor.java

mashhurs · 2025-07-10T22:27:30Z

src/main/java/co/elastic/logstash/filters/elasticintegration/EventProcessorBuilder.java

-                        extensions.add(new ConstantKeywordPainlessExtension()); // module: constant-keyword
-                        extensions.add(new ProcessorsWhitelistExtension());     // module: ingest-common
-                        extensions.add(new SpatialPainlessExtension());         // module: spatial
-                        extensions.add(new WildcardPainlessExtension());        // module: wildcard


I think these 4-extentions are missing ones.
When running integration tests, I get cannot resolve symbol [Processors] (below log is with a bit more context) error which I believe belong to this painless ProcessorsWhitelistExtension.

logstash-1 | Caused by: java.lang.IllegalArgumentException: cannot resolve symbol [Processors] logstash-1 | at org.elasticsearch.painless.PainlessScript$Script.compile(long bytes = Processors.bytes(params['size']); ctx.size_in_bytes = bytes;:24) ~[?:?] logstash-1 | at org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitCall(DefaultSemanticAnalysisPhase.java:3233)

It looks like we will need to find a way to apply #162 on the bridge-side in Elasticsearch.

mashhurs · 2025-07-10T22:28:57Z

src/main/java/co/elastic/logstash/filters/elasticintegration/IngestDuplexMarshaller.java

-
-        sourceAndMetadata.put(IngestDocument.Metadata.VERSION.getFieldName(), Objects.requireNonNullElse(sourceVersion, 1L));
+        // TODO: make IngestDocument.Metadata.VERSION.getFieldName() available
+        sourceAndMetadata.put("_version", Objects.requireNonNullElse(sourceVersion, 1L));


We need to wrap IngestDocument.Metadata as well. For now I wanted fully isolate plugin from the IngestDocument.

I would be in favor of adding a public static final class Constants or similar to the IngestDocumentBridge:

// public class IngestDocumentBridge extends StableBridgeAPI.Proxy<IngestDocument> { public static final class Constants { private Constants() {} public static final String METADATA_VERSION_FIELD_NAME = IngestDocument.Metadata.VERSION.getFieldName(); // ... } // }

mashhurs · 2025-07-10T22:30:26Z

src/main/java/co/elastic/logstash/filters/elasticintegration/geoip/GeoIpProcessorFactory.java

-        return new GeoIpProcessor.Factory("geoip", this.ipDatabaseProvider)
-                .create(processorFactories, tag, description, config, projectId);
+                            Map<String, Object> config) throws Exception {
+        return ProcessorBridge.wrap(new GeoIpProcessor.Factory("geoip", this.ipDatabaseProvider)


We will move rg.elasticsearch.ingest.geoip.GeoIpProcessor usage separately with GeoIP task.

src/main/java/co/elastic/logstash/filters/elasticintegration/ingest/PipelineProcessor.java

mashhurs · 2025-07-10T22:43:25Z

src/main/java/co/elastic/logstash/filters/elasticintegration/ingest/RedactPlugin.java

-import org.elasticsearch.ingest.Processor;
-import org.elasticsearch.plugins.IngestPlugin;
+import org.elasticsearch.logstashbridge.ingest.ProcessorBridge;
+import org.elasticsearch.logstashbridge.plugins.IngestPluginBridge;
 import org.elasticsearch.license.XPackLicenseState;


We may need to move all processors (including this source piece) to ES with next iterations.

mergify · 2025-07-10T23:14:21Z

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

src/main/java/co/elastic/logstash/filters/elasticintegration/PipelineConfigurationFactory.java

src/main/java/co/elastic/logstash/filters/elasticintegration/IngestDuplexMarshaller.java

mashhurs · 2025-07-23T06:25:11Z

gradle.properties

@@ -1,2 +1,3 @@
 LOGSTASH_PATH=../../logstash
-ELASTICSEARCH_TREEISH=main
+ELASTICSEARCH_REPO=mashhurs/elasticsearch
+ELASTICSEARCH_TREEISH=move-to-bridge-stable-api-investigation


temporary PoC, will be rolled back to origin once upstream PR is merged.

…s. Removes required upwrap interface since upstream moved to default method. Utilizes metadata version field since upstream made it accessible.

…sor) changes.

…nBridge

mashhurs · 2025-08-08T18:44:19Z

build.gradle

@@ -307,10 +307,6 @@ task shadeElasticsearchIngestGeoIpModule(type: com.github.jengelman.gradle.plugi

    mergeServiceFiles()

-    String shadeNamespace = "org.elasticsearch.ingest.geoip.shaded"
-    relocate('com.fasterxml.jackson', "${shadeNamespace}.com.fasterxml.jackson")
-    relocate('com.maxmind',           "${shadeNamespace}.com.maxmind")


We get "No X class found" exceptions if we relocate them and I am not seeing org.elasticsearch.ingest.geoip.shaded.com.fasterxml.jackson.* usages.

mashhurs · 2025-08-08T18:44:58Z

gradle.properties

@@ -1,2 +1,3 @@
 LOGSTASH_PATH=../../logstash
-ELASTICSEARCH_TREEISH=main
+ELASTICSEARCH_REPO=mashhurs/elasticsearch
+ELASTICSEARCH_TREEISH=logstash-bridge-geoip-interfaces


FYI: this will be removed once upstream PR is merged.

mashhurs · 2025-08-08T18:47:29Z

src/main/java/co/elastic/logstash/filters/elasticintegration/geoip/ValidatableIpDatabase.java

-
-import org.elasticsearch.ingest.geoip.IpDatabase;
-
-public interface ValidatableIpDatabase extends IpDatabase {


No direct usage.

mashhurs · 2025-08-08T18:48:06Z

src/main/java/co/elastic/logstash/filters/elasticintegration/ingest/RedactPlugin.java

-
-import java.util.Map;
-
-public class RedactPlugin extends Plugin implements IngestPlugin {


moved to the upstream bridge.

mashhurs · 2025-08-08T18:50:31Z

src/main/java/co/elastic/logstash/filters/elasticintegration/util/PluginProjectResolver.java

-import org.elasticsearch.cluster.project.ProjectResolver;
-import org.elasticsearch.core.CheckedRunnable;
-
-public class PluginProjectResolver implements ProjectResolver {


Moved to the bridge, ProjectIdResolverBridge class.

…s to logstash-bridge.

elasticmachine · 2025-08-11T20:05:07Z

💔 Build Failed

Buildkite Build
Commit: 2dda5e1

Failed CI Steps

History

💔 Build #641 failed 16dfbdb
💔 Build #640 failed 53ded41
💔 Build #639 failed fa3468c
💔 Build #638 failed 9b0b557

mashhurs commented Jul 10, 2025

View reviewed changes

mashhurs changed the base branch from move-to-es-bridge-api to main July 10, 2025 23:13

mashhurs added the backport-skip Skip automated backport with mergify label Jul 10, 2025

mashhurs force-pushed the move-to-es-bridge-investigation branch from 0707b1b to bc9af21 Compare July 10, 2025 23:16

mashhurs commented Jul 10, 2025

View reviewed changes

src/main/java/co/elastic/logstash/filters/elasticintegration/PipelineConfigurationFactory.java Show resolved Hide resolved

mashhurs changed the title ~~[Move to ES bridge API] Investigation~~ DO NOT MERGE: [Move to ES bridge API] Investigation Jul 10, 2025

mashhurs mentioned this pull request Jul 10, 2025

[Move to ES bridge -> ES scope] Evaluate current status of Stable API based on plugin usage #331

Closed

mashhurs commented Jul 16, 2025

View reviewed changes

src/main/java/co/elastic/logstash/filters/elasticintegration/IngestDuplexMarshaller.java Show resolved Hide resolved

mashhurs force-pushed the move-to-es-bridge-investigation branch from 18333c6 to 5a96c8c Compare July 17, 2025 20:06

mashhurs mentioned this pull request Jul 17, 2025

[Logstash] Move elastic_integration plugin usage to ES logstash-bridge. elastic/elasticsearch#131486

Merged

mashhurs changed the title ~~DO NOT MERGE: [Move to ES bridge API] Investigation~~ [DO NOT MERGE]: Move to ES bridge API Jul 18, 2025

mashhurs force-pushed the move-to-es-bridge-investigation branch from 9820699 to 9bdad09 Compare July 22, 2025 19:44

mashhurs changed the title ~~[DO NOT MERGE]: Move to ES bridge API~~ Move to ES bridge API Jul 23, 2025

mashhurs commented Jul 23, 2025

View reviewed changes

mashhurs linked an issue Jul 24, 2025 that may be closed by this pull request

[Move to ES Bridge -> plugin scope]: Move classes to Bridge API #333

Open

mashhurs mentioned this pull request Aug 5, 2025

Sync up with upstream ES main: apply project ID resolver to ScriptService. #357

Merged

mashhurs added 5 commits August 7, 2025 16:58

Replace ES classes with their Bridge Stable API pairs.

6c55a10

Points to personal ES repo which provides Stable API interface update…

52d8dd0

…s. Removes required upwrap interface since upstream moved to default method. Utilizes metadata version field since upstream made it accessible.

Plugins and processors moved to the bridge

294ae12

Apply conceptually modification (internal, external and bridge proces…

71cafcb

…sor) changes.

Utilize moved processor definition constants in the IngestCommonPlugi…

c2446a2

…nBridge

mashhurs force-pushed the move-to-es-bridge-investigation branch from 09c332c to 9b0b557 Compare August 7, 2025 23:58

Use logstash-bridge GeoIP interfaces.

53ded41

mashhurs force-pushed the move-to-es-bridge-investigation branch from fa3468c to 53ded41 Compare August 8, 2025 18:40

mashhurs commented Aug 8, 2025

View reviewed changes

mashhurs linked an issue Aug 8, 2025 that may be closed by this pull request

[Move to ES Bridge -> plugin scope]: Apply GeoIP changes in the plugins #354

Open

Apply interface changes in unit tests.

16dfbdb

mashhurs marked this pull request as ready for review August 11, 2025 16:51

Move RefCountingRunnable, FailProcessorException and Releasable usage…

2dda5e1

…s to logstash-bridge.

mashhurs linked an issue Aug 11, 2025 that may be closed by this pull request

[Move to ES bridge -> ES scope] Introduce wrappers for missed interfaces and close the gap #335

Open


		import org.elasticsearch.ingest.geoip.IpDatabase;

		public interface ValidatableIpDatabase extends IpDatabase {


		import java.util.Map;

		public class RedactPlugin extends Plugin implements IngestPlugin {

Move to ES bridge API #336

Are you sure you want to change the base?

Move to ES bridge API #336

Uh oh!

Conversation

mashhurs commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Next steps

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Jul 10, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Uh oh!

mashhurs commented Jul 10, 2025 •

edited

Loading

elasticmachine commented Aug 11, 2025 •

edited

Loading